Recap

  • What is relational data?
  • How to combine different data sets for data analysis?
  • Joins and Keys
  • Different types of joins
  • Stacking data frames
  • Working with Excel files

Outline

  1. What is web scraping?
  2. rvest and polite
  3. What is a function?
  4. File paths and RStudio projects

Web Scraping

Web Scraping

An increasing amount of data is available on the web, but they are often provided in an unstructured format.

Copying and pasting is always an option, but it’s time-consuming and prone to errors.

Web Scraping

Web scraping is the process of extracting data from websites automatically and transform it into a structured dataset.

There are two major types of web scraping:

Screen Scraping

Extracting data from the HTML content of a web page, with HTML parser or regular expression matching.

API (application programming interface) Data Retrieval

Website offers a set of structured http requests that return JSON or XML files.

R provides all the essential tools needed for web scraping, along with the advantage of direct data analysis. However, other languages like Python, Perl, and Java are also effective options.

Hypertext Markup Language (HTML)

HTML is the standard markup language for documents designed to be displayed in a web browser.

It defines the content and layout of web pages in a hierarchical, tree-like structure.

Here is a basic example of an HTML document:

<html>
  <head>
    <title>This is a title</title>
  </head>
  <body>
    <p style="font-size: 30px;">
      ETC1010
    </p>
  </body>
</html>

Hypertext Markup Language (HTML)

An HTML document is made up of HTML elements.

An HTML element consists of:

  • a start tag (e.g. <p>)
  • optional attributes (e.g. style="...")
  • an end tag (e.g. </p>)
  • contents, which include everything between the start and end tags
<html>
  <head>
    <title>This is a title</title>
  </head>
  <body>
    <p style="font-size: 30px;">
      ETC1010
    </p>
  </body>
</html>

There are over 100 HTML elements.

  • <html>: indicates the content is HTML code.
  • <head>: meta information about the document.
  • <title>: title of the document.
  • <body>: visible content of the webpage.
  • <p>: a paragraph/a block of text.

rvest

rvest

The rvest R package offers a suite of tools for extracting information from HTML documents:

  • read_html(): Read an HTML file.
  • html_elements(): Select elements from the HTML document using CSS selectors.
  • html_text2(): Extract the plain text contents of an HTML element.
  • html_attrs(): Get element attributes.
  • read_html_live(): runs a live web browser (Chrome) in the background and read an HTML file that are generated dynamically by JavaScript.
  • …

read_html()

We typically start the scraping process with read_html(), which makes a request to the website and then reads the HTML code of the page.

  • Get the HTML document of IMDB top 250 movies:
library(rvest)
read_html("https://www.imdb.com/chart/top/")
{html_document}
<html lang="en-US" xmlns:og="http://opengraphprotocol.org/schema/" xmlns:fb="http://www.facebook.com/2008/fbml">
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
[2] <body>\n<div>    <img height="1" width="1" style="display:none;visibility ...

Tip

read_html() works for most websites but can be problematic on sites where the text we aim to extract is dynamically generated by JavaScript.

html_elements()

html_elements() finds HTML elements using CSS selectors.

It helps us select specific parts of the HTML file that we are interested in.

  • Get the <head> element:
read_html("https://www.imdb.com/chart/top/") %>%
  html_elements("head")
{xml_nodeset (1)}
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
  • Get the <title> from the <head> element:
read_html("https://www.imdb.com/chart/top/") %>%
  html_elements("head") %>%
  html_elements("title")
{xml_nodeset (1)}
[1] <title>IMDb Top 250 Movies</title>\n

html_text2()

html_text2() extracts text from HTML elements.

  • Get the text out of the title:
read_html("https://www.imdb.com/chart/top/") %>%
  html_elements("head") %>%
  html_elements("title") %>%
  html_text2()
[1] "IMDb Top 250 Movies"
  • Get the text out of all the <p>:
read_html("https://www.imdb.com/chart/top/") %>%
  html_elements("p") %>%
  html_text2()
[1] "The Top Rated Movie list only includes feature films."
[2] "© 1990- 2024 by IMDb.com, Inc."

html_attrs()

html_attrs() gets all attributes of HTML elements.

  • Get the attributes out of all the <p>:
read_html("https://www.imdb.com/chart/top/") %>%
  html_elements("p") %>%
  html_attrs()
[[1]]
named character(0)

[[2]]
                                     class 
"imdb-footer__copyright footer__copyright"

Note

The first <p> element has no attributes, so it returns an empty result.

CSS Selector

CSS Selector

CSS selectors help you pinpoint the exact parts of a webpage you want to scrape, such as text, images, or links.

Here’s a cheatsheet to guide you:

Selector Example Description
element p Select all <p> elements
element element div p Select all <p> elements inside a <div> element
element > element div > p Select all <p> elements with <div> as a parent
.class .title Select all elements with class="title"
#id .name Select all elements with id="name"
[attribute] [class] Select all elements with a class attribute
[attribute=value] [class=title] Select all elements with class="title"
element:first-child div:first-child Select first child inside a <div> element

SelectorGadget

SelectorGadget allows you to click on elements within a webpage and see the corresponding selectors that match those elements.

Install the Chrome Extension.

  • It does not work well on websites with dynamically generated content (e.g. IMDB)!

Usage

  • A box will open in the bottom right of the website. Click on a page element that you would like your selector to match (it will turn green). SelectorGadget will then generate a minimal CSS selector for that element, and will highlight (yellow) everything that is matched by the selector.
  • Now click on a highlighted element to remove it from the selector (red), or click on an unhighlighted element to add it to the selector. Through this process of selection and rejection, SelectorGadget helps you come up with the appropriate CSS selector for your needs.

Google Chrome DevTools

  1. You can right-click on any element on a webpage and choose “Inspect” to open DevTools.

  2. In DevTools, click on the top-left icon that looks like a square with a cursor inside.

  3. Hover over any part of the webpage, a tooltip will appear, showing the CSS selector.

Top 250 Movies on IMDB

Let’s get titles of top 250 movies.

Each title is in an <h3> element with the class ipc-title__text.

read_html("https://www.imdb.com/chart/top/") %>%
  html_elements("h3.ipc-title__text") %>%
  html_text2() -> raw_title
raw_title
 [1] "IMDb Charts"                                         
 [2] "1. The Shawshank Redemption"                         
 [3] "2. The Godfather"                                    
 [4] "3. The Dark Knight"                                  
 [5] "4. The Godfather: Part II"                           
 [6] "5. 12 Angry Men"                                     
 [7] "6. Schindler's List"                                 
 [8] "7. The Lord of the Rings: The Return of the King"    
 [9] "8. Pulp Fiction"                                     
[10] "9. The Lord of the Rings: The Fellowship of the Ring"
[11] "10. The Good, the Bad and the Ugly"                  
[12] "11. Forrest Gump"                                    
[13] "12. The Lord of the Rings: The Two Towers"           
[14] "13. Fight Club"                                      
[15] "14. Inception"                                       
[16] "15. Star Wars: Episode V - The Empire Strikes Back"  
[17] "16. The Matrix"                                      
[18] "17. GoodFellas"                                      
[19] "18. One Flew Over the Cuckoo's Nest"                 
[20] "19. Interstellar"                                    
[21] "20. Se7en"                                           
[22] "21. It's a Wonderful Life"                           
[23] "22. Seven Samurai"                                   
[24] "23. The Silence of the Lambs"                        
[25] "24. Saving Private Ryan"                             
[26] "25. City of God"                                     
[27] "Recently viewed"

Note

There are only 25 movies because the content on the webpage is dynamically generated by JavaScript.

read_html_live()

read_html_live() runs a live web browser (Chrome) in the background.

It allows you to access dynamically generated HTML elements and interact with the live page, such as clicking buttons or typing in forms.

read_html_live("https://www.imdb.com/chart/top/") %>%
  html_elements("h3.ipc-title__text") %>%
  html_text2() -> raw_title
raw_title
  [1] "IMDb Charts"                                                             
  [2] "1. The Shawshank Redemption"                                             
  [3] "2. The Godfather"                                                        
  [4] "3. The Dark Knight"                                                      
  [5] "4. The Godfather Part II"                                                
  [6] "5. 12 Angry Men"                                                         
  [7] "6. Schindler's List"                                                     
  [8] "7. The Lord of the Rings: The Return of the King"                        
  [9] "8. Pulp Fiction"                                                         
 [10] "9. The Lord of the Rings: The Fellowship of the Ring"                    
 [11] "10. The Good, the Bad and the Ugly"                                      
 [12] "11. Forrest Gump"                                                        
 [13] "12. The Lord of the Rings: The Two Towers"                               
 [14] "13. Fight Club"                                                          
 [15] "14. Inception"                                                           
 [16] "15. Star Wars: Episode V - The Empire Strikes Back"                      
 [17] "16. The Matrix"                                                          
 [18] "17. GoodFellas"                                                          
 [19] "18. One Flew Over the Cuckoo's Nest"                                     
 [20] "19. Interstellar"                                                        
 [21] "20. Seven"                                                               
 [22] "21. It's a Wonderful Life"                                               
 [23] "22. Seven Samurai"                                                       
 [24] "23. The Silence of the Lambs"                                            
 [25] "24. Saving Private Ryan"                                                 
 [26] "25. City of God"                                                         
 [27] "26. Life Is Beautiful"                                                   
 [28] "27. The Green Mile"                                                      
 [29] "28. Terminator 2: Judgment Day"                                          
 [30] "29. Star Wars: Episode IV - A New Hope"                                  
 [31] "30. Back to the Future"                                                  
 [32] "31. Spirited Away"                                                       
 [33] "32. The Pianist"                                                         
 [34] "33. Parasite"                                                            
 [35] "34. Psycho"                                                              
 [36] "35. Gladiator"                                                           
 [37] "36. The Lion King"                                                       
 [38] "37. Spider-Man: Across the Spider-Verse"                                 
 [39] "38. The Departed"                                                        
 [40] "39. Whiplash"                                                            
 [41] "40. American History X"                                                  
 [42] "41. Leon"                                                                
 [43] "42. Grave of the Fireflies"                                              
 [44] "43. The Prestige"                                                        
 [45] "44. Harakiri"                                                            
 [46] "45. Dune: Part Two"                                                      
 [47] "46. The Usual Suspects"                                                  
 [48] "47. Casablanca"                                                          
 [49] "48. Untouchable"                                                         
 [50] "49. Cinema Paradiso"                                                     
 [51] "50. Modern Times"                                                        
 [52] "51. Alien"                                                               
 [53] "52. Rear Window"                                                         
 [54] "53. Once Upon a Time in the West"                                        
 [55] "54. City Lights"                                                         
 [56] "55. Django Unchained"                                                    
 [57] "56. Apocalypse Now"                                                      
 [58] "57. Memento"                                                             
 [59] "58. WALL·E"                                                              
 [60] "59. Raiders of the Lost Ark"                                             
 [61] "60. 12th Fail"                                                           
 [62] "61. The Lives of Others"                                                 
 [63] "62. Sunset Blvd."                                                        
 [64] "63. Avengers: Infinity War"                                              
 [65] "64. Paths of Glory"                                                      
 [66] "65. Spider-Man: Into the Spider-Verse"                                   
 [67] "66. Witness for the Prosecution"                                         
 [68] "67. The Shining"                                                         
 [69] "68. The Great Dictator"                                                  
 [70] "69. Aliens"                                                              
 [71] "70. Inglourious Basterds"                                                
 [72] "71. The Dark Knight Rises"                                               
 [73] "72. Coco"                                                                
 [74] "73. Amadeus"                                                             
 [75] "74. Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb"
 [76] "75. Toy Story"                                                           
 [77] "76. Oldboy"                                                              
 [78] "77. American Beauty"                                                     
 [79] "78. Avengers: Endgame"                                                   
 [80] "79. Braveheart"                                                          
 [81] "80. Das Boot"                                                            
 [82] "81. Good Will Hunting"                                                   
 [83] "82. Princess Mononoke"                                                   
 [84] "83. Joker"                                                               
 [85] "84. Your Name."                                                          
 [86] "85. High and Low"                                                        
 [87] "86. 3 Idiots"                                                            
 [88] "87. Once Upon a Time in America"                                         
 [89] "88. Singin' in the Rain"                                                 
 [90] "89. Capernaum"                                                           
 [91] "90. Come and See"                                                        
 [92] "91. Requiem for a Dream"                                                 
 [93] "92. Toy Story 3"                                                         
 [94] "93. Star Wars: Episode VI - Return of the Jedi"                          
 [95] "94. Eternal Sunshine of the Spotless Mind"                               
 [96] "95. The Hunt"                                                            
 [97] "96. 2001: A Space Odyssey"                                               
 [98] "97. Ikiru"                                                               
 [99] "98. Reservoir Dogs"                                                      
[100] "99. Lawrence of Arabia"                                                  
[101] "100. The Apartment"                                                      
[102] "101. Incendies"                                                          
[103] "102. Oppenheimer"                                                        
[104] "103. North by Northwest"                                                 
[105] "104. Scarface"                                                           
[106] "105. Citizen Kane"                                                       
[107] "106. Double Indemnity"                                                   
[108] "107. M"                                                                  
[109] "108. Vertigo"                                                            
[110] "109. Full Metal Jacket"                                                  
[111] "110. Heat"                                                               
[112] "111. Amélie"                                                             
[113] "112. Up"                                                                 
[114] "113. A Clockwork Orange"                                                 
[115] "114. To Kill a Mockingbird"                                              
[116] "115. A Separation"                                                       
[117] "116. The Sting"                                                          
[118] "117. Die Hard"                                                           
[119] "118. Indiana Jones and the Last Crusade"                                 
[120] "119. Like Stars on Earth"                                                
[121] "120. Metropolis"                                                         
[122] "121. Snatch"                                                             
[123] "122. 1917"                                                               
[124] "123. L.A. Confidential"                                                  
[125] "124. Bicycle Thieves"                                                    
[126] "125. Hamilton"                                                           
[127] "126. Taxi Driver"                                                        
[128] "127. Downfall"                                                           
[129] "128. Dangal"                                                             
[130] "129. Batman Begins"                                                      
[131] "130. For a Few Dollars More"                                             
[132] "131. The Wolf of Wall Street"                                            
[133] "132. Green Book"                                                         
[134] "133. Some Like It Hot"                                                   
[135] "134. The Kid"                                                            
[136] "135. Judgment at Nuremberg"                                              
[137] "136. The Truman Show"                                                    
[138] "137. The Father"                                                         
[139] "138. All About Eve"                                                      
[140] "139. Shutter Island"                                                     
[141] "140. There Will Be Blood"                                                
[142] "141. Top Gun: Maverick"                                                  
[143] "142. Jurassic Park"                                                      
[144] "143. Casino"                                                             
[145] "144. Ran"                                                                
[146] "145. The Sixth Sense"                                                    
[147] "146. Pan's Labyrinth"                                                    
[148] "147. Unforgiven"                                                         
[149] "148. No Country for Old Men"                                             
[150] "149. The Thing"                                                          
[151] "150. A Beautiful Mind"                                                   
[152] "151. Kill Bill: Vol. 1"                                                  
[153] "152. The Treasure of the Sierra Madre"                                   
[154] "153. Yojimbo"                                                            
[155] "154. The Great Escape"                                                   
[156] "155. Monty Python and the Holy Grail"                                    
[157] "156. Finding Nemo"                                                       
[158] "157. Prisoners"                                                          
[159] "158. Howl's Moving Castle"                                               
[160] "159. Rashomon"                                                           
[161] "160. The Elephant Man"                                                   
[162] "161. Dial M for Murder"                                                  
[163] "162. Chinatown"                                                          
[164] "163. Gone with the Wind"                                                 
[165] "164. Lock, Stock and Two Smoking Barrels"                                
[166] "165. The Secret in Their Eyes"                                           
[167] "166. Inside Out"                                                         
[168] "167. V for Vendetta"                                                     
[169] "168. Raging Bull"                                                        
[170] "169. Three Billboards Outside Ebbing, Missouri"                          
[171] "170. Trainspotting"                                                      
[172] "171. The Bridge on the River Kwai"                                       
[173] "172. Klaus"                                                              
[174] "173. Catch Me If You Can"                                                
[175] "174. Fargo"                                                              
[176] "175. Warrior"                                                            
[177] "176. Spider-Man: No Way Home"                                            
[178] "177. Gran Torino"                                                        
[179] "178. Harry Potter and the Deathly Hallows: Part 2"                       
[180] "179. Million Dollar Baby"                                                
[181] "180. My Neighbour Totoro"                                                
[182] "181. Mad Max: Fury Road"                                                 
[183] "182. Children of Heaven"                                                 
[184] "183. Ben-Hur"                                                            
[185] "184. 12 Years a Slave"                                                   
[186] "185. Before Sunrise"                                                     
[187] "186. Blade Runner"                                                       
[188] "187. Barry Lyndon"                                                       
[189] "188. The Grand Budapest Hotel"                                           
[190] "189. Hacksaw Ridge"                                                      
[191] "190. Gone Girl"                                                          
[192] "191. Dead Poets Society"                                                 
[193] "192. Maharaja"                                                           
[194] "193. Memories of Murder"                                                 
[195] "194. In the Name of the Father"                                          
[196] "195. The Gold Rush"                                                      
[197] "196. Monsters, Inc."                                                     
[198] "197. Wild Tales"                                                         
[199] "198. The Deer Hunter"                                                    
[200] "199. The General"                                                        
[201] "200. Jaws"                                                               
[202] "201. Sherlock Jr."                                                       
[203] "202. How to Train Your Dragon"                                           
[204] "203. Ratatouille"                                                        
[205] "204. On the Waterfront"                                                  
[206] "205. Mary and Max"                                                       
[207] "206. The Wages of Fear"                                                  
[208] "207. The Third Man"                                                      
[209] "208. Wild Strawberries"                                                  
[210] "209. Le Mans '66"                                                        
[211] "210. Mr. Smith Goes to Washington"                                       
[212] "211. Tokyo Story"                                                        
[213] "212. Logan"                                                              
[214] "213. Rocky"                                                              
[215] "214. The Big Lebowski"                                                   
[216] "215. The Seventh Seal"                                                   
[217] "216. Room"                                                               
[218] "217. Spotlight"                                                          
[219] "218. The Terminator"                                                     
[220] "219. Hotel Rwanda"                                                       
[221] "220. Platoon"                                                            
[222] "221. La haine"                                                           
[223] "222. Pirates of the Caribbean: The Curse of the Black Pearl"             
[224] "223. Before Sunset"                                                      
[225] "224. The Passion of Joan of Arc"                                         
[226] "225. Jai Bhim"                                                           
[227] "226. The Best Years of Our Lives"                                        
[228] "227. The Exorcist"                                                       
[229] "228. Rush"                                                               
[230] "229. The Incredibles"                                                    
[231] "230. Network"                                                            
[232] "231. The Wizard of Oz"                                                   
[233] "232. Stand by Me"                                                        
[234] "233. Hachi: A Dog's Tale"                                                
[235] "234. The Sound of Music"                                                 
[236] "235. My Father and My Son"                                               
[237] "236. The Handmaiden"                                                     
[238] "237. To Be or Not to Be"                                                 
[239] "238. Into the Wild"                                                      
[240] "239. The Battle of Algiers"                                              
[241] "240. Groundhog Day"                                                      
[242] "241. The Grapes of Wrath"                                                
[243] "242. The Iron Giant"                                                     
[244] "243. Amores perros"                                                      
[245] "244. Rebecca"                                                            
[246] "245. The Help"                                                           
[247] "246. Cool Hand Luke"                                                     
[248] "247. It Happened One Night"                                              
[249] "248. Paris, Texas"                                                       
[250] "249. Aladdin"                                                            
[251] "250. Scent of a Woman"                                                   
[252] "You have rated"                                                          
[253] "More to explore"                                                         
[254] "Charts"                                                                  
[255] "Top Box Office (US)"                                                     
[256] "Most Popular Movies"                                                     
[257] "Top Rated English Movies"                                                
[258] "Most Popular TV Shows"                                                   
[259] "Top 250 TV Shows"                                                        
[260] "Lowest Rated Movies"                                                     
[261] "Most Popular Celebs"                                                     
[262] "Movie News"                                                              
[263] "Top Rated Movies by Genre"                                               
[264] "Recently viewed"

Note

Since we’re using a headless browser to access the website, it will execute the JavaScript and load all the dynamic content, similar to how you would see it when visiting the site with a standard browser.

Parsing the Raw Data

  1. Find all real titles by matching strings that start with (^) a digit (\d) using a regular expression.

  2. Match the pattern of starting digit(s), followed by a dot and a space, and replace them with an empty string.

Regular expressions are a concise and flexible tool for describing patterns in strings. See here for more.

Talk much more about these next week.

Complete Workflow to Get Title, Rating and Year

doc <- read_html_live("https://www.imdb.com/chart/top/")
title <- doc %>%
  html_elements("h3.ipc-title__text") %>%
  html_text2() %>%
  str_subset("^\\d") %>%
  str_replace("^\\d+\\. ", "")
rating <- doc %>%
  html_elements("span.ipc-rating-star--rating") %>%
  html_text2()
year <- doc %>%
  html_elements(".sc-b189961a-7.btCcOY.cli-title-metadata > :first-child") %>%
  html_text2()

tibble(title, rating, year)
# A tibble: 250 × 3
   title                                             rating year 
   <chr>                                             <chr>  <chr>
 1 The Shawshank Redemption                          9.3    1994 
 2 The Godfather                                     9.2    1972 
 3 The Dark Knight                                   9.0    2008 
 4 The Godfather Part II                             9.0    1974 
 5 12 Angry Men                                      9.0    1957 
 6 Schindler's List                                  9.0    1993 
 7 The Lord of the Rings: The Return of the King     9.0    2003 
 8 Pulp Fiction                                      8.9    1994 
 9 The Lord of the Rings: The Fellowship of the Ring 8.9    2001 
10 The Good, the Bad and the Ugly                    8.8    1966 
# ℹ 240 more rows

polite

Responsible Web Scraping

The polite package helps ensure responsible web scraping by following polite practices:

  • introducing yourself
  • asking for permission
  • scraping slowly
  • not making repeated requests

Responsible Web Scraping

Use bow() to introduce yourself to the website and check the robots.txt file to get permission to scrape.

library(polite)
session <- bow("http://www.imdb.com")
session
<polite session> http://www.imdb.com
    User-agent: polite R package
    robots.txt: 35 rules are defined for 3 bots
   Crawl delay: 5 sec
  The path is scrapable for this user-agent

scrape() works similarly to read_html().

scrape(session)
{html_document}
<html lang="en-US" xmlns:og="http://opengraphprotocol.org/schema/" xmlns:fb="http://www.facebook.com/2008/fbml">
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
[2] <body>\n<div>    <img height="1" width="1" style="display:none;visibility ...

Ohter Common Formats: JSON

Data are sometimes stored as JavaScript Object Notation (JSON), which requires special unpacking by the jsonlite package.

jsonlite::fromJSON(
'[{
    "Name": "Mario",
    "Age": 32,
    "Occupation": "Plumber"
  },
  {
    "Name": "Peach",
    "Age": 21,
    "Occupation": "Princess"
  },
  {},
  {
    "Name": "Bowser",
    "Occupation": "Koopa"
  }]')
    Name Age Occupation
1  Mario  32    Plumber
2  Peach  21   Princess
3   <NA>  NA       <NA>
4 Bowser  NA      Koopa

Potential Challenges with Web Scraping

  • Unreliable formatting at the source
  • Data broken into many pages
  • Data arriving in multiple excel file formats
  • …
  • We will come back to this after we learn about functions in today’s class!

Further Exploring

People write R packages to access online data! Check out:

Let’s have a break!

Function

Function

A function is a block of organized, reusable code that performs a specific task.

Read this introduction to functions.

Here is an example:

Function Components

  • Formals: the list of arguments which controls how you can call the function
  • Body: the code inside the function

Why Functions?

Automate common tasks in a powerful and general way:

  • You can give a function an evocative name that makes your code easier to understand.

  • As requirements change, you only need to update code in one place, instead of many.

  • You eliminate the chance of making incidental mistakes when you copy and paste (i.e. updating a variable name in one place, but not in another).

  • If you need to do the same thing more than twice in an analysis, you should write a function for it.

Web Scraping For TV Series

We want to find out how many episodes a TV series listed on IMDb has.

stranger_things <- bow("http://www.imdb.com/title/tt4574334/") %>%
  scrape()
the_last_of_us <- bow("https://www.imdb.com/title/tt3581920") %>%
  scrape()

How Many Episodes in TV Series?

How Many Episodes in Stranger Things?

There are multiple <span> elements with the class ipc-title__subtext.

We need the first one.

stranger_things %>%
  html_elements("span.ipc-title__subtext") %>%
  html_text2() %>%
  .[1] %>%
  as.integer()
[1] 42

How Many Episodes in The Last of Us?

The webpage layout is the same for TV series, so the code is nearly identical.

the_last_of_us %>%
  html_elements("span.ipc-title__subtext") %>%
  html_text2() %>%
  .[1] %>%
  as.integer()
[1] 16

When Should You Write A Function?

  • Whenever you’ve copied and pasted a block of code more than twice.
  • When you want to clearly express some set of actions
  • When you want to do modular coding.
  • There are many other reasons as well!

Let’s Make A Function to Get the Number of Episodes

  1. We first need to pick a short but informative name, preferably a verb.
get_episode_number <- function(...) {
  ...
}
  1. List inputs, or arguments, to the function inside function.
get_episode_number <- function(series_doc) {
  ...
}
  1. Place the code you have developed in body of the function ({}).
get_episode_number <- function(series_doc) {
  series_doc %>%
    html_elements("span.ipc-title__subtext") %>%
    html_text2() %>%
    .[1] %>%
    as.integer()
}

Check Your Function

  • Number of episodes in The Walking Dead
the_walking_dead <- bow("http://www.imdb.com/title/tt1520211") %>%
  scrape()
get_episode_number(the_walking_dead)
[1] 177
  • Number of episodes in Breaking Bad
breaking_bad <- bow("https://www.imdb.com/title/tt0903747") %>%
  scrape()
get_episode_number(breaking_bad)
[1] 62

How to Update the Function to Use Page URL as Argument?

We can retrieve the HTML document within the function body.

get_episode_number <- function(series_url) {
  series_url %>%
    bow() %>%
    scrape() %>%
    html_elements("span.ipc-title__subtext") %>%
    html_text2() %>%
    .[1] %>%
    as.integer()
}
  • Number of episodes in The Wire
the_wire_url <- "https://www.imdb.com/title/tt0306414"
get_episode_number(the_wire_url)
[1] 60

Let’s Update the Function to Also Get TV Series Names

This function returns a tibble.

get_episode_number <- function(series_url) {
  series_doc <- series_url %>%
    bow() %>%
    scrape()
  
  eps_number <- series_doc %>%
    html_elements("span.ipc-title__subtext") %>%
    html_text2() %>%
    .[1] %>%
    as.integer()
  
  series_name <- series_doc %>%
    html_elements("span.hero__primary-text") %>%
    html_text2()
  
  return(tibble(name = series_name,
                episode_number = eps_number))
}

Number of episodes in The Wire

get_episode_number(the_wire_url)
# A tibble: 1 × 2
  name     episode_number
  <chr>             <int>
1 The Wire             60

Automation

Automation

  • You now have a function that scrapes the name and episode number for TV shows given their URLs.
  • Where can we find a list of URLs for the top 100 most popular TV shows on IMDb?

Automation

All the TV show links are within elements that have the class ipc-title-link-wrapper.

We need to extract the href attribute from the <a> elements to obtain the URLs.

Need to use read_html_live() because of the dynamic content.

series_urls <- read_html_live("http://www.imdb.com/chart/tvmeter") %>%
  html_elements("a.ipc-title-link-wrapper") %>%
  html_attr("href") %>%
  str_subset("title") %>%
  paste0("http://www.imdb.com", .)
series_urls
  [1] "http://www.imdb.com/title/tt7631058/?ref_=chttvm_t_1"  
  [2] "http://www.imdb.com/title/tt8550732/?ref_=chttvm_t_2"  
  [3] "http://www.imdb.com/title/tt0944947/?ref_=chttvm_t_3"  
  [4] "http://www.imdb.com/title/tt15203646/?ref_=chttvm_t_4" 
  [5] "http://www.imdb.com/title/tt11691774/?ref_=chttvm_t_5" 
  [6] "http://www.imdb.com/title/tt11198330/?ref_=chttvm_t_6" 
  [7] "http://www.imdb.com/title/tt1312171/?ref_=chttvm_t_7"  
  [8] "http://www.imdb.com/title/tt0411008/?ref_=chttvm_t_8"  
  [9] "http://www.imdb.com/title/tt1190634/?ref_=chttvm_t_9"  
 [10] "http://www.imdb.com/title/tt14452776/?ref_=chttvm_t_10"
 [11] "http://www.imdb.com/title/tt0903747/?ref_=chttvm_t_11" 
 [12] "http://www.imdb.com/title/tt0773262/?ref_=chttvm_t_12" 
 [13] "http://www.imdb.com/title/tt15017118/?ref_=chttvm_t_13"
 [14] "http://www.imdb.com/title/tt14153236/?ref_=chttvm_t_14"
 [15] "http://www.imdb.com/title/tt7671070/?ref_=chttvm_t_15" 
 [16] "http://www.imdb.com/title/tt7587890/?ref_=chttvm_t_16" 
 [17] "http://www.imdb.com/title/tt2177461/?ref_=chttvm_t_17" 
 [18] "http://www.imdb.com/title/tt30068042/?ref_=chttvm_t_18"
 [19] "http://www.imdb.com/title/tt8962124/?ref_=chttvm_t_19" 
 [20] "http://www.imdb.com/title/tt9813792/?ref_=chttvm_t_20" 
 [21] "http://www.imdb.com/title/tt2708480/?ref_=chttvm_t_21" 
 [22] "http://www.imdb.com/title/tt4236770/?ref_=chttvm_t_22" 
 [23] "http://www.imdb.com/title/tt26767508/?ref_=chttvm_t_23"
 [24] "http://www.imdb.com/title/tt1520211/?ref_=chttvm_t_24" 
 [25] "http://www.imdb.com/title/tt17677860/?ref_=chttvm_t_25"
 [26] "http://www.imdb.com/title/tt4574334/?ref_=chttvm_t_26" 
 [27] "http://www.imdb.com/title/tt0452046/?ref_=chttvm_t_27" 
 [28] "http://www.imdb.com/title/tt0141842/?ref_=chttvm_t_28" 
 [29] "http://www.imdb.com/title/tt0460681/?ref_=chttvm_t_29" 
 [30] "http://www.imdb.com/title/tt0455275/?ref_=chttvm_t_30" 
 [31] "http://www.imdb.com/title/tt5875444/?ref_=chttvm_t_31" 
 [32] "http://www.imdb.com/title/tt13994572/?ref_=chttvm_t_32"
 [33] "http://www.imdb.com/title/tt9055008/?ref_=chttvm_t_33" 
 [34] "http://www.imdb.com/title/tt15591076/?ref_=chttvm_t_34"
 [35] "http://www.imdb.com/title/tt0413573/?ref_=chttvm_t_35" 
 [36] "http://www.imdb.com/title/tt9257258/?ref_=chttvm_t_36" 
 [37] "http://www.imdb.com/title/tt10986410/?ref_=chttvm_t_37"
 [38] "http://www.imdb.com/title/tt0203259/?ref_=chttvm_t_38" 
 [39] "http://www.imdb.com/title/tt0412142/?ref_=chttvm_t_39" 
 [40] "http://www.imdb.com/title/tt3032476/?ref_=chttvm_t_40" 
 [41] "http://www.imdb.com/title/tt2356777/?ref_=chttvm_t_41" 
 [42] "http://www.imdb.com/title/tt1632701/?ref_=chttvm_t_42" 
 [43] "http://www.imdb.com/title/tt1586680/?ref_=chttvm_t_43" 
 [44] "http://www.imdb.com/title/tt12637874/?ref_=chttvm_t_44"
 [45] "http://www.imdb.com/title/tt0386676/?ref_=chttvm_t_45" 
 [46] "http://www.imdb.com/title/tt15428778/?ref_=chttvm_t_46"
 [47] "http://www.imdb.com/title/tt14022350/?ref_=chttvm_t_47"
 [48] "http://www.imdb.com/title/tt2560140/?ref_=chttvm_t_48" 
 [49] "http://www.imdb.com/title/tt10569934/?ref_=chttvm_t_49"
 [50] "http://www.imdb.com/title/tt33038128/?ref_=chttvm_t_50"
 [51] "http://www.imdb.com/title/tt9174582/?ref_=chttvm_t_51" 
 [52] "http://www.imdb.com/title/tt0364845/?ref_=chttvm_t_52" 
 [53] "http://www.imdb.com/title/tt0898266/?ref_=chttvm_t_53" 
 [54] "http://www.imdb.com/title/tt2306299/?ref_=chttvm_t_54" 
 [55] "http://www.imdb.com/title/tt0108778/?ref_=chttvm_t_55" 
 [56] "http://www.imdb.com/title/tt1442437/?ref_=chttvm_t_56" 
 [57] "http://www.imdb.com/title/tt2788316/?ref_=chttvm_t_57" 
 [58] "http://www.imdb.com/title/tt7660850/?ref_=chttvm_t_58" 
 [59] "http://www.imdb.com/title/tt2085059/?ref_=chttvm_t_59" 
 [60] "http://www.imdb.com/title/tt8888462/?ref_=chttvm_t_60" 
 [61] "http://www.imdb.com/title/tt2442560/?ref_=chttvm_t_61" 
 [62] "http://www.imdb.com/title/tt0804503/?ref_=chttvm_t_62" 
 [63] "http://www.imdb.com/title/tt2802850/?ref_=chttvm_t_63" 
 [64] "http://www.imdb.com/title/tt0118401/?ref_=chttvm_t_64" 
 [65] "http://www.imdb.com/title/tt3581920/?ref_=chttvm_t_65" 
 [66] "http://www.imdb.com/title/tt4786824/?ref_=chttvm_t_66" 
 [67] "http://www.imdb.com/title/tt12262202/?ref_=chttvm_t_67"
 [68] "http://www.imdb.com/title/tt6226232/?ref_=chttvm_t_68" 
 [69] "http://www.imdb.com/title/tt0306414/?ref_=chttvm_t_69" 
 [70] "http://www.imdb.com/title/tt13406094/?ref_=chttvm_t_70"
 [71] "http://www.imdb.com/title/tt6473344/?ref_=chttvm_t_71" 
 [72] "http://www.imdb.com/title/tt0388629/?ref_=chttvm_t_72" 
 [73] "http://www.imdb.com/title/tt0460649/?ref_=chttvm_t_73" 
 [74] "http://www.imdb.com/title/tt13210838/?ref_=chttvm_t_74"
 [75] "http://www.imdb.com/title/tt3960394/?ref_=chttvm_t_75" 
 [76] "http://www.imdb.com/title/tt21831910/?ref_=chttvm_t_76"
 [77] "http://www.imdb.com/title/tt28118211/?ref_=chttvm_t_77"
 [78] "http://www.imdb.com/title/tt5016504/?ref_=chttvm_t_78" 
 [79] "http://www.imdb.com/title/tt11280740/?ref_=chttvm_t_79"
 [80] "http://www.imdb.com/title/tt0460627/?ref_=chttvm_t_80" 
 [81] "http://www.imdb.com/title/tt7221388/?ref_=chttvm_t_81" 
 [82] "http://www.imdb.com/title/tt8740790/?ref_=chttvm_t_82" 
 [83] "http://www.imdb.com/title/tt16098700/?ref_=chttvm_t_83"
 [84] "http://www.imdb.com/title/tt0436992/?ref_=chttvm_t_84" 
 [85] "http://www.imdb.com/title/tt0238784/?ref_=chttvm_t_85" 
 [86] "http://www.imdb.com/title/tt0106028/?ref_=chttvm_t_86" 
 [87] "http://www.imdb.com/title/tt1844624/?ref_=chttvm_t_87" 
 [88] "http://www.imdb.com/title/tt2861424/?ref_=chttvm_t_88" 
 [89] "http://www.imdb.com/title/tt11712058/?ref_=chttvm_t_89"
 [90] "http://www.imdb.com/title/tt0472954/?ref_=chttvm_t_90" 
 [91] "http://www.imdb.com/title/tt19231492/?ref_=chttvm_t_91"
 [92] "http://www.imdb.com/title/tt8772296/?ref_=chttvm_t_92" 
 [93] "http://www.imdb.com/title/tt3006802/?ref_=chttvm_t_93" 
 [94] "http://www.imdb.com/title/tt0106179/?ref_=chttvm_t_94" 
 [95] "http://www.imdb.com/title/tt0491738/?ref_=chttvm_t_95" 
 [96] "http://www.imdb.com/title/tt16358384/?ref_=chttvm_t_96"
 [97] "http://www.imdb.com/title/tt13443470/?ref_=chttvm_t_97"
 [98] "http://www.imdb.com/title/tt1928307/?ref_=chttvm_t_98" 
 [99] "http://www.imdb.com/title/tt14022668/?ref_=chttvm_t_99"
[100] "http://www.imdb.com/title/tt2467372/?ref_=chttvm_t_100"

Go to Each Page, Scrape Information

Programatically direct R to each page on the series_urls vector and run get_episode_number().

get_episode_number(series_urls[1])
# A tibble: 1 × 2
  name                                      episode_number
  <chr>                                              <int>
1 The Lord of the Rings: The Rings of Power             17
get_episode_number(series_urls[2])
# A tibble: 1 × 2
  name  episode_number
  <chr>          <int>
1 Kaos               8
get_episode_number(series_urls[3])
# A tibble: 1 × 2
  name            episode_number
  <chr>                    <int>
1 Game of Thrones             74

Go to Each Page, Scrape Information

In other words, we want to map the get_episode_number() function to each element of series_urls.

This will hit the URLs one after another, and grab the information.

map_df() will combine all the tibbles into one.

map_df(series_urls, get_episode_number) %>%
  head()
# A tibble: 5 × 2
  name                                      episode_number
  <chr>                                              <int>
1 The Lord of the Rings: The Rings of Power             17
2 Kaos                                                   8
3 Game of Thrones                                       74
4 Bad Monkey                                            10
5 Only Murders in the Building                          50

File Paths and Project Organization

Please read this section on your own time

File Paths and Project Organization

  • It’s important when you start working on your own machine that you understand file storage hygiene.
  • It helps prevent unexpected problems and makes you more productive
  • You’ll spend less time fighting against strange file paths.
  • Not sure what a file path is? We will explain that as well!

What is A File Path?

  • This all might be a bit confusing if you don’t know what a file path is.
  • A file path is: “the machine-readable directions to where files on your computer live.”
  • So, this file path:
/Users/starwars/rmd4sci-materials/demo-gapminder.Rmd

Describes the location of the file “demo-gapminder.Rmd”.

What is A File Path?

We could visualize this path:

/Users/starwars/rmd4sci-materials/demo-gapminder.Rmd

as:

users
 └── starwars
     └── rmd4sci-materials
         └── demo-gapminder.Rmd

To read in the gapminder.csv file, you might need to write code like this:

gapminder <- read_csv("/Users/starwars/Desktop/rmd4sci-materials/data/gapminder.csv"

This is a problem, because this is not portable code!

Using setwd()

Read this great blog post written by Jenny Bryan.

Sometimes this is the first line of an R Script or R markdown file.

setwd("c:/really/long/file/path/to/this/directory)

What do you think the setwd() code does?

  • “Set my working directory to this specific working directory”.

It means that you can read in data and other things like this:

data <- read_csv("data/mydata.csv")

Instead of

data <- read_csv("c:/really/long/file/path/to/this/directory/data/mydata.csv")

Using setwd()

  • This has the effect of making the file paths work in your file
  • This is a problem because, among other things, using setwd():
    • Has 0% chance of working on someone else’s machine (this includes you in >6 months)
    • Your file is not self-contained and portable. (Think: “What if this folder moved to /Downloads, or onto another machine?”)
  • To get this to work, you need to hand edit the file path to your machine.
  • This is painful. And when you do this all the time, it gets old, fast.
  • setwd() is banned by CRAN in package development!

What is the Alternative to setwd()?

I highly recommend when you start on a new idea, new research project, paper. Anything that is new. It should start its life as an RStudio project.

An RStudio project helps keep related work together in the same place. Amongst other things, they:

  • Keep all your files together
  • Set the working directory to the project directory
  • Starts a new session of R
  • Restore previously edited files into the editor tabs
  • Restore other rstudio settings
  • Allow for multiple R projects open at the same time.

RStudio Project

This helps keep you sane, because:

  • Your projects are each independent.
  • You can work on different projects at the same time.
  • Objects and functions you create and run from project idea won’t impact one another.
  • You can refer to your data and other projects in a consistent way.

And finally, the big one: RStudio projects help resolve file path problems, because they automatically set the working directory to the location of the RStudio project.

here

In some cases you might have many folders in your RStudio project.

To help navigate them appropriately, you can use the here package to provide the full path directory, in a compact way.

Get the absolute path to the data folder:

here::here("data")
[1] "/Users/patrickli/Desktop/rproject/ida2024s2/data"

Get the absolute path to the gapminder.csv:

here::here("data", "gapminder.csv")
[1] "/Users/patrickli/Desktop/rproject/ida2024s2/data/gapminder.csv"

You can read the above here code as: In the folder data of the current RStudio project, there is a file called gapminder.csv, can you please give me the absolute path to that file?

here

This is really handy for a few reasons:

  1. It makes things completely portable
  2. Rmarkdown/Quarto documents have a special way of looking for files, this helps eliminate file path pain.
  3. If you decide to not use RStudio projects, you have code that will work on any machine.

ETC1010/ETC5510 Lecture 8 | Melbourne time